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The Evaluation of the Enhanced Positive School Climate Model 


Purpose. The Santa Ana Unified School District received an i3 grant (Investing in Innovation; funded by 
the U.S. Department of Education Grant number U411C160074) to build on their existing school climate 
practices. Called the Enhanced Positive School Climate Model, the aim is to improve school climate, 
student-adult relationships, create social emotional learning programs, and provide students the 
needed structure to access challenging curriculum and expectations. As part of this enhanced model, 
School Climate Liaisons (SCLs) were hired to support and provide coaching to schools in the intervention 
group on PBIS, RP implementation, and behavioral interventions. As part of the i3 grant requirements, 
the district employed a rigorous cluster-randomized trial whereby half of the district’s elementary 
schools were assigned to receive the Enhanced Positive School Climate Model and half were assigned to 
a waitlist control. The confirmatory research question is: 


Do students in grades 4 and 5 at follow-up, in schools assigned to receive support from PBIS and 
Restorative Practice school climate specialists and community liaisons, who receive services for 

two years, exhibit higher levels of self-management at the end of the second year, as compared 
to students in grades 4 and 5 in schools assigned to not receive such support? 


Additionally, there were exploratory research questions examining changes in other SEL measures 
including growth mindset, self-efficacy, and social awareness. 


Methods. This study employed a cluster-randomized controlled design whereby half of the study 
schools were assigned to the treatment condition and the other half of schools were assigned to the 
“business as usual” control condition. In total, 35 schools (30 K-5 schools; 5 K-8 schools) were randomly 
assigned to either the treatment or control condition. All students within these schools were included in 
the evaluation. Students in the treatment schools (n = 17) were eligible to receive services through the 
grant. Randomization occurred in May 2017. The 2016-17 school year served as a baseline school year 
(i.e., no services were provided). The Enhanced Positive School Climate Model began in the 2017-18 
school year and continued into the 2018-19 school year for treatment schools only. Control schools 
were waitlisted and began receiving services in the 2019-20 school year. Students enrolled in district 
schools in May 2017 were tracked using student roster information through the end of the 2018-19 
school year. Only students who were in study schools continuously from the 2016-2017 to 2018-2019 
school years were included in the impact analysis. The confirmatory and exploratory outcomes 
referenced below were conducted using extant data from the district. Data were used from routine data 
collection processes within the district from the annual CORE climate student survey. The primary 
outcome measure was the five-item self-management scale on the CORE District’s student survey 
(Taylor, Oberle, Durlak, & Weissberg, 2017; Transforming Education, 2014; 
https://www.rand.org/education-and-labor/projects/assessments/tool/2014/panorama-social- 
emotional-learning-questionnaire-measures.html) asked of all students in grades 4-12 each year. For the 
confirmatory research question (and all exploratory questions), adjusted post-intervention outcomes for 
students in treatment schools were compared to the outcomes for their counterparts in the control 
schools. This involved fitting conditional multilevel regression models (i.e., hierarchical linear modeling 
{[HLM]), with additional terms to account for the nesting of individuals within schools (see Goldstein 
1987; Raudenbush & Bryk, 2002; Murray 1998). 


Results. Eight two-level models were estimated per grade level to address the confirmatory and 
exploratory research questions. Although the students in treatment schools had slightly higher self- 
management, growth mindset and self-efficacy scores at follow-up, only the difference in fourth grade 
self-efficacy scores was Statistically significant. In contrast, students in control schools had very slightly 
higher social awareness scores; this finding was not statistically significant. 


Implications. Meaningful and statistically significant differences were not found between students who 
attended schools that received the Enhanced Positive School Climate Model compared to their peers in 
schools that did not receive the Enhanced Positive School Climate Model. The lack of findings may be 
due to the other changes in school climate practices throughout the district during the study period. 
Additionally, variations in the implementation of the Enhanced Positive School Climate Model may have 
contributed to the lack of findings. Although the findings are not significant, the direction of results is 
mostly consistent, indicating increases in SEL competencies for students receiving additional school 
climate supports. Future studies should continue to investigate the relationship between whole-school 
approaches to school climate and changes in SEL competencies for students. 


The Evaluation of the Enhanced Positive 
School Climate Model 


The purpose of Santa Ana Unified School District’s i3 (Investing in Innovation; funded by the U.S. 
Department of Education Grant number U411C160074) proposal was to build on their existing school 
climate practices -- called the Enhanced Positive School Climate Model (PSCM). Santa Ana Unified School 
District (SAUSD) is the largest school district in Orange County and seventh largest in California. At the 
start of the project, the district was comprised of 54 schools, serving over 53,000 students, and ranked 
2™ in the state for student population density. Based on the National Free and Reduced Lunch Program 
(FRLP), 91 percent of SAUSD students qualified for FRLP, an indicator of child poverty. SAUSD had 82 
percent of its students categorized as English Learners (EL), making it the number one port of entry for 
new EL students in the country. 


Santa Ana is the most populous city in the county and the 8" most densely populated city in the United 
States. Second and third generation families live under the same roof, many of whom live in converted 
garages or rooms not meant for human occupancy. The Department of Housing and Urban Development 
(HUD) designated four Santa Ana residential areas as Empowerment Zones, or in severe and immediate 
need for intervention in the areas of economic development, health care, housing, childcare, and 
transportation. In Santa Ana, families often work multiple minimum wage paying jobs with long hours, 
yet lack the finances to provide essentials, such as consistent shelter, food, and clothing for the family. 


In 2008, SAUSD started to focus on building a positive school culture and climate with its initial roll-out 
of Positive Behavioral Intervention and Supports (PBIS). By the 2015-16 school year, district-wide efforts 
to train and implement Positive Behavioral Intervention and Supports (PBIS) and Restorative Practice 
(RP) were in place. At the start of this grant, all school sites within the district were participating in 
trainings for PBIS and RP based on their differentiating levels of implementation. By infusing restorative 
practices, an alternative to punitive practices, within the PBIS framework, SAUSD has been able to 
monitor instructional, behavioral, and evidence-based intervention practices for students with its 
Enhanced Positive School Climate Model (Enhanced PSCM). 


Reporting on Intervention and Comparison Conditions 


Description of Project Intervention Activities 


The Enhanced PSCM’s aim was to improve school climate, student-adult relationships, create social 
emotional learning programs, and provide students the needed structure to access challenging 
curriculum and expectations. As part of this enhanced model, School Climate Liaisons (SCLs) were hired 
to support and provide coaching to schools in the intervention group on PBIS, RP implementation, and 
behavioral interventions. While schools assigned to both intervention and control groups, implemented 
PBIS and RP, schools assigned to the intervention group were provided additional supports from the 
SCLs. The additional supports provided by SCLs included coaching school site staff, implementation 
support, and school climate enhancement activities implemented by the school climate liaisons. 


The focus of this evaluation was to examine the extent to which providing these additional supports 
improved the effectiveness of SAUSD’s existing PSCM. The theory of change for the Enhanced PSCM was 
that -- by providing personnel and resources; organizational support and internal expertise; input from 


parents and the community; by providing coaching and trainings to school staff and administrators on 
PBIS and RP tools and behavioral interventions; by training parents on PBIS, RP tools, and behavioral 


interventions; and by connecting families to resources to meet their needs -- there would be increases in 


students’ social emotional skills (including self-management), a reduction in office discipline referrals 


and suspensions, and an increase in parent awareness of PBIS and RP strategies. 


The Logic Model provides an overview of SAUSD’s PSCM, the Enhanced PSCM funded by i3, and the 
short- and long-term outcomes (Exhibit 1). The key components of the Enhanced PSCM, as outlined in 


the logic model, are: 


provide coaching to intervention school sites on Tiers 1-3, 


provide training to parents on PBIS, RP tools, and behavioral interventions, 


provide parent trainings on PBIS, RP strategies, behavioral interventions, 


conduct home visits with families, and 


assist families by connecting them to resources. 


Exhibit 1: Logic Model: Santa Ana Unified School District’s Enhanced PSCM 


Positive School Enhanced - Short-term Outcomes Long-term Outcomes 
Climate Model Positive School (Mediators) 
Climate Model 
Key Components 


Provide training to 
Positive Behavioral 
Interventions & 
Supports (PBIS) 
leadership on School- 
wide Information 
System (SWIS). 


Implement the use of 
SWIS. 


Provide Tier 2 and 3 
services to students. 


Provide trainings on 
behavioral 
interventions to school 
staff. 


Provide coaching to 

school sites on 

interventions for 

Tiers 1-3. Coaching 

examples include: 

a. Fidelity of PBIS 
practices 

b. Implementation 
of RP tools (e. g. 
conferences, 
circles) 

c. Collecting and 
monitoring data 


Provide trainings to 
parents on PBIS and 
RP Tools and 
behavioral 
interventions. Parent 
trainings include 


The successful 

implementation of the 

Positive School Climate 

Model (PSCM) will result 

in: 

e Increase measure of 
fidelity of the model. 


e Increase networking of 
SPLCs between schools 
to share ideas about 
PSCM. 


e Greater use of SWIS for 
documenting data by 
school coordinators at 
target sites. 


e Increased staff 
awareness of 


Students in target schools 

will improve SEL and well- 

being. The successful 

implementation of the 

PSCM results in: 

e Anincrease of student 
self-management 
skills. 


e More students 
exhibiting more self- 
efficacy and growth 
mindset skills. 


e Areduction in rate of 
office discipline 
referrals. 


e Areduction in student 
suspension rate. 


Implement Restorative 
Conferences with youth 
(Tier 3). 


Host Data Dialogues. 


Provide trainings to 
counselors, school 
psychologists, teachers, 
administrators, and to 
PBIS leadership teams 
on PBIS, restorative 
practice (RP) tools, and 
behavioral 
interventions. 


Parent School Climate 
Institutes. 


Conduct home visits 
with families. 


Assist parents with 
identifying resources 
to meet student/ 
family needs. 


behavioral 
interventions. 


Parent attendance at 
RP, PBIS or Parent 
School Climate 
Institute Trainings. 


Increase in parents’ 
awareness of PBIS and 
RP strategies. 


Streamline 
communication with 
community partners 


e Anincrease in student 
academic achievement. 


The successful 
implementation of the 
enhanced PSCM will result 
in teachers/staff having 
greater knowledge, skills, 
and practice of PBIS, RP, 
and behavioral 
interventions. 


The successful 
implementation of the 
enhanced PSCM will 
nurture an attendance 
going culture at schools to 
promote school 
connectedness. 


Content, Duration, and Intensity of Key Activities and Intensity 


The key elements of the Enhanced PSCM are focused on two systems — PBIS and RP-— in place within the 


district. While these systems were already established before the i3, there was variation in the level of 


implementation/adoption at each school. There were schools that had adopted these processes several 
years before the i3 project was funded, while other sites had started implementing these systems a year 
before the start of this grant. Below is a description of PBIS and RP: 


1. Positive Behavioral Intervention and Supports provides a layered continuum of supports for the 
students in the district. Every student receives universal supports such as school-wide and 
classroom expectations, mental health awareness, use of affective statements, and positive 
reinforcement systems (Tier 1). Some students also receive targeted supports such as social 
skills, self-management, mentoring, and case management (Tier 2). Using the PBIS framework, 
only a few students receive intensive supports such as intensive social skills teaching and 
individual behavior management plans (Tier 3). 


Two of the key Tier 2 strategies included the Coordination of Services Teams (COSTs) and Check 
In, Check Out (CICO). COST is a multidisciplinary team of school staff and providers who 
collaborate on linking referred students to resources and interventions. The goal of COST is to 
provide an avenue for identifying the needs of individual students, creating linkages for 
students, and supporting students’ academic success and healthy development. CICO is another 
Tier 2 support based on intervention that focuses on checking in with specific students to 


monitor goals and provide encouragement. 


2. Restorative practices at SAUSD focus on the belief that relationships within schools are central 


components of school climate. RP includes the practice of affective statements, restorative 


questions, small impromptu conversations for de-escalating problems quickly, structured circles 
designed to strengthen and repair relationships, and conferences that bring people together to 
discuss serious concerns and discipline issues. 


District staff facilitated professional learning activities related to PBIS and RP in all schools in the district 
as part of the business as usual professional learning. This included classroom management, 
interventions, assessments, SWIS, and RP. The treatment sites had the additional support of SCLs at 
their site one day per week. Each school had a designated school climate liaison. SCLs provided 
substantive coaching to personnel on their PBIS approach, on organizing evidence-based practices, 
improving the implementation of those practices, on how to implement RP strategies, and COST 
meeting practices. SCLs provided multiple types of coaching support for school sites that focused on 
strategies such as conference circles and the collection and monitoring of data to address students’ 
needs. Additionally, SCLs supported the target school sites with their PBIS Tiered Fidelity Inventory 
Survey (TFI) completion and provided consultations with staff on referring students to school-based and 
community-based services. 


The type of coaching provided was differentiated based on the needs of the school. In some schools 
they provided demonstrations of how to lead restorative practices such as classroom circles, mediation, 
and conferences. In other sites, they co-facilitated circles with staff trained in RP strategies. SCLs also 
encouraged school staff at treatment sites to attend RP training provided by the district in an effort to 
strengthen sustainability of practices and fidelity of RP practices at the sites. 


Parent trainings were provided by School Climate Liaisons at the intervention sites. The topics for the 
parent trainings were differentiated based on school site needs but mostly focused on parenting 
strategies, behavioral interventions, and restorative practice tools. SCLs also facilitated restorative 
practice circles with parents. School Climate Parent Institutes were held annually and only parents from 
treatment sites were invited to attend the institutes. School Climate Liaisons worked with each 
intervention site to recruit a designated number of parents to participate in the institutes. The goal of 
the institutes was to provide in-depth training to a select group of parents from each treatment site on 
PBIS, RP, social-emotional learning, and graduation requirements. SAUSD held a pilot School Climate 
Institute for parents in June 2017. Based on feedback received from the pilot, additional Institutes were 
held each school year. Parents attending the institute completed approximately a 10-hour course over a 
series of weeks. SCLs also provided resources and linkages to families to community resources through 
parent meetings and home visits. The resources provided were based on the needs of students and 
families. 


Objectives of key activities 


The Enhanced PSCM tracked progress on the following goals and objectives over the course of the grant: 
Goal 1: Improve social-emotional learning (SEL) and well-being of students in target schools 


Objective 1.1: School sites participating in the Positive School Climate Model will reduce the 
rate of office discipline referrals (ODRs) by 15 percent by spring 2019 as measured by 


referral data in SWIS (school-wide information systems) and District Student Information 
System (AERIES). 


Objective 1.2: School sites participating in the Positive School Climate Model will reduce the 
rate of suspensions by 25 percent by the spring 2019 as measured by AERIES. 


Objective 1.3: At least 75% of students in participating schools will report an increase, by 
spring 2019, in self-management as measured by the California Office of Reforming 
Education (CORE SEL) Survey. 


Objective 1.4: The overall attendance rate at school sites participating in the Positive School 
Climate Model will improve by .5 percent by school level by spring 2019 as measured by 
AERIES. 


Goal 2: Increase teacher/staff knowledge, skills, and practice of PBIS, RP, and behavioral interventions 


Objective 2.1: School sites participating in the Positive School Climate Model that also 
received a Tiered Fidelity Inventory (TFI) score below 70 percent will increase their score to 
at least 85 percent by the end of the school year as measured by the District PBIS team 
members. 


Goal 3: Increase parent awareness of PBIS and RP strategies to improve student behavior and 
engagement 


Objective 3.1: At least 5% of parents based on school ADA from participating schools will 
attend trainings on PBIS, RP, or Parent School Climate Institute by spring 2019 as measured 
by training logs 


Resources/training/coaching provided to individuals delivering the intervention 


The Program Specialist and School Climate Liaisons hired at the beginning of grant participated in a 
series of trainings over the course of several weeks. The Program Specialist was a certificated staff 
member who had been an existing district employee. The SCLs included both certificated and classified 
staff and their onboarding included trainings on the following: 


RP: Certified district trainers provided a 2-day RP training that included circle scenarios and 
how to coach school staff in integrating RP practices in their classroom. The SCLs also 
received annual RP refresher trainings. 


PBIS: Three of the initial SCLs hired for the grant, were already PBISApps certified. Training on 
the PBIS framework was provided by PBIS-trained facilitators. SCLs were also trained in 
administering the Tiered Fidelity Inventory (TFI) to measure the fidelity of PBIS. The 
training was a PBIS multi-day training focused on each Tier: 


Tier 1: 6-hour training. 
Tier 2 Part A: 6-hour training that focused on systems and included COST. 
Tier 2 Part B: 2-hour training that focused on Check In, Check Out. 


Tier 3: 2-day training that focused on behavior assessments and interventions. 


Coaching: The Program Specialist and SCLs participated in Cognitive Coaching Series training 
provided by an outside provider. 


Procedural/Site Level Administrator: SCLs participated in a training provided by district staff 
that focused on processes and strategies to collaborate with the administrators. In 
addition, the training included content on how to support sites in integrating services. 


Other: SCLs were trained on required district policies and procedures including child abuse 
reporting and the school district’s data and reporting system. 


Each SCL was shadowed by the i3 Coordinator and Program Specialist during the roll-out period of the 
grant. Over the course of the grant, some turnover occurred with SCLs. Trainings were provided to new 
SCLs when new staff were hired. New staff were also shadowed after completion of their on-boarding 
training. 


The Independent Evaluation of the Enhanced School Climate Model 


WestEd was contracted to provide an independent evaluation of the project. WestEd conducted the 
randomization, impact analysis, and reporting independent of the grantee. Data collection relied on 
routine, extant data collected by the school district on an annual basis and was not implemented 
specifically for this project. The confirmatory evaluation question is: 


Do students in grades 4 and 5 at follow-up, in schools assigned to receive support from PBIS and 
Restorative Practice school climate specialists and community liaisons, who receive services for two 
years, exhibit higher levels of self-management at the end of the second year, as compared to students 
in grades 4 and 5 in schools assigned to not receive such support? 


Additionally, there were exploratory research questions examining changes in other SEL measures 
including growth mindset, self-efficacy, and social awareness. 


The remainder of the report discusses WestEd’s evaluation of the Enhanced Positive School Climate 
Model. 


The Participants in the Intervention 


This study employed a cluster-randomized controlled design whereby half of the study schools were 
assigned to the treatment condition and the other half of schools were assigned to the “business as 
usual” control condition. The study design and randomization processes are discussed in more detail in 
the Study Design and Samples section. 


The eligibility criteria for inclusion in the study were elementary schools serving students in grades 2-5 
because the confirmatory evaluation question focuses on students who were in grades 4 and 5 at 
follow-up (i.e., students were in grades 2 and 3 at baseline). This included all elementary schools in the 
district except for one school because they only served students K-3. The evaluation focuses on grades 4 
and 5 because there are not enough schools serving grades 6-12 in the district to reliably detect impacts 
for school types serving these grade levels. 


In total, 35 schools (30 K-5 schools; 5 K-8 schools) were randomly assigned to either the treatment or 
control condition. All students within these schools were included in the evaluation. Students in the 
treatment schools (n = 17) were eligible to receive services through the grant. 


Santa Ana Unified School District serves high proportions of students at risk of failing academically. At 
the time of the grant application, approximately 91 percent of students in the district qualified for 
eligibiblity for free/reduced-price meals, and 82 percent of students were categorized as EL students. 


Randomization occurred in May 2017. The 2016-17 school year served as a baseline school year (i.e., no 
services were provided). The Enhanced Positive School Climate Model began in the 2017-18 school year 
and continued into the 2018-19 school year for treatment schools only. Control schools were waitlisted 
and began receiving services in the 2019-20 school year. 


Students in treatment schools received the Enhanced PSCMD. The Enhanced Positive School Climate 
model included multiple tiers of services as described in Description of Project Intervention Activities 
Section. 


The analytic sample did not include joiners. Students enrolled in district schools in May 2017 (the point 
of school randomization) were tracked using student roster information through the end of the 2018-19 
academic year. Only students who were in study schools continuously from May 2017 to May 2019 were 
included in the impact analysis. 


Students were analyzed as part of the group they were assigned to at baseline (e.g., if they were ina 
treatment school at baseline, but then transferred to another school, they were analyzed as part of the 
treatment group). To preserve the integrity of random assignment, and account for self-selection, 
outcome analyses included all randomly assigned schools, regardless of whether they actually received 
support from school climate specialists and community liaisons. 


During the study years (i.e., 2017-18-2018-19), the control schools received business-as-usual. 


Study Design and Sample 


Schools were randomized in May 2017 using two nested blocking variables. Schools were first stratified 
by grade level (i.e., K-5, K-8), and then based on aggregated grade-specific academic test scores within 
each school grade-level stratum. In elementary schools (K-5 grades), the average of 3 grade math and 
reading Smarter Balanced scale scores was used to determine randomization strata, whereby the school 
average scores were ranked and placed into 4 strata based on quartiles. In K-8 schools, 3 and 6" grade 
math and reading Smarter balanced scale scores were averaged, and schools were placed into two 
strata based on the median of school-specific averages across the 5 schools. The randomization resulted 
in the following distribution of schools, within school type and randomization strata, assigned to 
treatment and control conditions. The strata are included in Figure 2. 


Exhibit 2. Randomization Stratum and Study Condition 


Study Condition | Study Condition 


School Type Stratum Control Treatment Total Schools 
K-5 1 4 3 7 
K-5 2 4 8 
K-5 3 4 4 8 
K-5 4 3 4 7 
K-8 5 1 1 2 
K-8 6 2 1 3 
Total Schools by 18 17 35 
Condition 


WestEd’s lead analyst who was unfamiliar with the study schools conducted the randomizations using 
Stata. In instances where there were an odd number of schools within a stratum (e.g., stratum 1), 
assignment to treatment or control alternated. Every school had a 50 percent chance of assignment to 
either condition. All schools were randomly assigned in May 2017. 


Following random assignment, no changes were made to the schools included in the study. The student 
sample included students who were in grades 2 and 3 at baseline and were thus in grades 4 and 5 at 
follow-up. Students’ school membership was captured using attendance records from the 2017-18 
school year. Students’ school membership was tracked each subsequent year using the same 
attendance records data source. 


Baseline and Outcome Measures 


Students enrolled in district schools in May 2017 were tracked using student roster information through 
the end of the 2018-19 school year. Only students who were in study schools continuously from the 
2016-2017 to 2018-2019 school years were included in the impact analysis. The confirmatory and 
exploratory outcomes referenced below were conducted using extant data from the district. Data was 
used from routine data collection processes within the district from the annual CORE student survey. 
Thus, the evaluators were not involved in data collection. 


Measures 


The primary outcome measure was the five-item? self-management scale on the CORE District’s student 
survey (Taylor, Oberle, Durlak, & Weissberg, 2017’; Transforming Education, 2014": 
https://www.rand.org/education-and-labor/projects/assessments/tool/2014/panorama-social- 
emotional-learning-questionnaire-measures.html) asked of all students in grades 4-12 each year. The 
CORE student survey is a survey developed and implemented by the CORE Districts of California, which 
are situated in Fresno, Garden Grove, Long Beach, Los Angeles, Oakland, Sacramento, San Francisco, and 
Santa Ana. The items ask how often in the past 30 days students regulated emotions and behaviors (e.g., 
| stayed calm even when others bothered or criticized me”), with responses ranging from 1 (almost 
never) to 5 (almost all of the time). The scale has previously been shown to have good internal reliability 


1 Note that this measure was implemented as a 9-item measure in the 2016-17 school year, but only a 5-item measure in the 2018- 
19 school year. 


and WestEd found good reliability for the current sample (See Figure 3). Over the years of project 
implementation this scale was reduced to five items from the original nine. For the impact analysis, the 
five-item scale was used for baseline and year 2 measures. Panorama Education has demonstrated face 
validity of the construct. The CORE student survey has been shown to be a good indicator of changes in 
social and emotional learning. 


Exploratory measures from the same CORE student survey included: Growth mindset, self-efficacy, and 
social awareness (See Figure 3). Similar to the self-management measure, the social awareness items 
were reduced to five items from the original eight. The table below displays the reliability statistics from 
the 2016-17 and 2018-19 survey administrations. The CORE subscales and alphas are displayed in Figure 
3. 


The CORE student survey is administered to students in grades 4-12, thus no student survey data was 
available at baseline when students were in grades 2 and 3. As a result, the baseline measure of the 
outcomes for the CORE student survey scales for grades 4 (for the grade 2 analytic model) and 5 (for the 
grade 3 analytic model) were imputed using the school level averages of the respective outcomes for 4" 
and 5" graders at baseline. 


Figure 3. CORE Subscales, Items, and Alpha for Current Study 


N 
CORE Subscale a. 2016-17 Coefficient Alpha | 2018-19 Coefficient Alpha 
Self-Management 5 0.75 0.77 
Growth Mindset 4 0.70 0.83 
Self-efficacy 4 0.86 0.88 
Social Awareness 5 0.72 0.73 


Note. Self-management was a 9-item scale in 2016-17 school year, and a 5-item scale in the 2018-19 school year. The alpha shown reflects 
the same five items across school years for the current study. 


Data were extracted from the district’s existing database at baseline (Spring 2017) and follow-up 
(Summer 2019). 


Student-level demographic characteristics were acquired from the district at baseline. These 
demographics included gender, race and ethnicity, English language learner designation, and free 
and/or reduced-price lunch status. 


Reporting on Analytic Approach 


For the confirmatory question (and all exploratory questions), adjusted post-intervention outcomes for 
students in treatment schools were compared to the outcomes for their counterparts in the control 
schools. This involved fitting conditional multilevel regression models (i.e., hierarchical linear modeling 
{[HLM]), with additional terms to account for the nesting of individuals within schools (see Goldstein 
1987"". Raudenbush & Bryk, 2002"; Murray 1998"). The study involved school-level randomization and 


delivery of additional supports at the school level. The design thus involved clustering at the school 
level, as students and teachers are nested within schools. The random effect of school was included in 
the models to account for the nesting of observations within schools. The fixed effects included in the 
models were treatment group, baseline (pretest) measures of outcome variables, randomization strata, 
and vectors of aggregated student-level covariates. The purpose of including statistical controls was to 
minimize random error and to increase the precision of the estimates. 


The following type of two-level HLM, in reduced form, for a continuous outcome serves as an example 
of the model. 


Selfmgmtj; = A + B,Preselfmgmt; + B2Tx; + UBilij + tj + ei [1] 


Where subscripts i and / denote student and school, respectively; Se/fmgmt represents the CORE survey 
5-item self-management scale, measured at the end of the second implementation year; Preselfmgmt 
represents the baseline measure of the outcome variable aggregated at the school-level; Tx is a 
dichotomous variable indicating student attendance at the school assigned to the treatment condition; 
and / is a vector of other control variables for students and a set of dichotomous variables for all but one 
of the randomization strata. Lastly, t; represent random variables for schools (clustering groups), and Ejx 
is an error term for individual sample members. In this model, the intervention effect is represented by 
B2TX;, which captures the regression adjusted treatment/control school differences in the outcome 
variable. t; capture random effects (intercepts) of school, which account for the positive intraclass 
correlations in the data. 


The above HLM model was employed for the models of continuous confirmatory and exploratory 
outcomes of interest. Analysis of each outcome of interest was conducted separately by grade level. 


Missing Data Approach 


Student records with missing values for the outcome of interest and baseline demographic covariates 
were not included in the analytic samples. 


Baseline Equivalence and Attrition 


First, WestEd assessed cluster-level (schools) attrition. Because no schools dropped out of the study, 
there was no cluster-level attrition. Second, WestEd assessed student-level attrition. For students in 
grade 4 at follow-up, attrition was 24 percent of students (3,493 students were in schools at 
randomization; only 2,586 students remained at follow-up). The differential attrition between the 
treatment and control groups was about 1.8 percent. For students in grade 5 at follow-up, attrition was 
26 percent of students (3,934 students were in schools at randomization; only 2,994 students remained 
at follow-up). The differential attrition between the treatment and control groups was about 0.7 
percent. Thus, overall and differential attrition were both low (What Works Clearinghouse v 4.0, 2020) 
for both grades indicating a tolerable threat of bias under both optimistic and cautious assumptions. 
Because attrition was low and an RCT was implemented, and there was no need to assess baseline 
equivalence. 


Reporting on Study Results 


Attrition 


Figure 4 displays the attrition for the grades 2 and 3 students randomized at baseline through the 2-year 
follow-up period. On average, for all outcome measures there were about 75 percent of grade 4 
students and 76 percent of fifth grade students from the treatment group retained through the study 
period. There were 73 percent of grade 4 students and 76 percent of grade 5 students from the 
comparison group retained through the study period. This made for 1.8 percent differential attrition for 
the grade 4 analytic sample and 0.7 percent differential attrition for the grade 5 analytic sample. 


Figure 4, Student and cluster level attrition 
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Eight two-level models were estimated per grade level to address the confirmatory and exploratory 
research questions. The figures below detail the results from the unadjusted post-test means for the 
comparison groups and model-adjusted post-test means for the treatment groups (Figures 7-10). The 


adjusted models were computed using the outcome of interest, treatment status, strata, and the 
covariates of gender, race/ethnicity, English language proficiency, free/reduced lunch status and 
baseline measure of the outcome of interest, all clustered at the school level. Additionally, model- 
adjusted mean differences were estimated for the outcomes and standardized differences were 
computed using Hedge’s g. 


Self-Management 

The self-management scale comprises five Likert scale items. Responses to these items were selected 
from an interval scale of 1 to 5 with 1 representing more negative perceptions of self-management and 
5 representing more positive perceptions of self-management. For each student, responses to the five 
Likert scale items were inclusively averaged (meaning all non-missing responses amongst the five items 
were averaged). 


The adjusted model results for grade 4 show both the treatment (MM = 3.88) and comparison group (M = 
3.86) have more positive leaning neutral perceptions of self-management. Though not statistically 
significant the treatment group did show a 0.02 higher self-management score than the comparison 
group after two years of participating in the intervention. The grade 5 models show that the treatment 
and comparison group both showed positive perceptions of self-management. There was virtually no 
difference in the treatment (M = 3.94) and comparison (M = 3.945) groups’ self-management scores. 
Figure 7 shows the results for the grades 4 and 5 self-management outcomes. 


Figure 7. Self-Management Outcomes Estimates 
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Growth Mindset 

The growth mindset scale is comprised of four Likert scale items. Responses to these items were 
selected from an interval scale of 1 to 5 with 1 representing more negative perceptions of growth 
mindset and 5 representing more positive perceptions of growth mindset. For each student, responses 


to the four Likert scale items were inclusively averaged (meaning all non-missing responses amongst the 
four items were averaged). 


The adjusted model results for grade 4 show both the treatment (M = 3.97) and comparison group (M = 
3.91) have positive perceptions of growth mindset. Though not statistically significant, the treatment 
group did show a .06-point higher growth mindset score than the comparison group after two years of 
participating in the intervention. The grade 5 models show positive perceptions for growth mindset for 
both treatment (M = 4.06) and comparison groups (M = 4.02). The treatment group exhibited scores 
that were 0.04-points higher than the comparison group, on average. There was no statistically 
significant difference. Figure 8 shows the estimates for grades 4 and 5 for growth mindset outcomes. 


Figure 8. Growth Mindset Outcomes Estimates 


Comparison Group Treatment Group 
Sample Size Sample Size Mod Treatme 
el nt Standardi p- 
Control zed valu 
Differen Difference e 
ce 


n n n 
cluste studen cluste studen 
rs ts rs ts 


4" Grade 


Growth 
Mindse 16 1,253 3.91 ; 17 1,332 3.97 a 0.057 0.062 pe 
t 


5 Grade 
Growth 
- : 0.7 0.33 


Mindse 17 1,510 4.06 j 0.042 0.048 3 
t 


Notes: Adju= Adjusted estimates included covariates: stratum, baseline outcome, gender, ethnicity, English language fluency, and FRL 
status. 


Self-Efficacy 

The self-efficacy scale is composed of four Likert scale items. Responses to these items were selected 
from an interval scale of 1 to 5 with 1 representing more negative perceptions of self-efficacy and 5 
representing more positive perceptions of self-efficacy. For each student, responses to the four Likert 
scale items were inclusively averaged (meaning all non-missing responses amongst the four items were 
averaged). 


The adjusted model results for grade 4 students show both the treatment (IM = 3.64) and comparison 
group (M = 3.52) have more neutral perceptions of self-efficacy. The treatment group displayed a 0.12- 
point higher self-efficacy score than the comparison group on average after two years of participating in 
the intervention (p = .027). The grade 5 models showed positive leaning neutral perceptions of self- 
efficacy for both treatment and comparison groups. There was no difference in the groups’ self-efficacy 
scores (M = 3.57). Figure 9 shows the self-efficacy outcome estimates for both grade levels. 


Figure 9. Self-Efficacy Outcomes Estimates 
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Social Awareness 

The social awareness scale is comprised of five Likert scale items. Responses to these items were 
selected from an interval scale of 1 to 5 with 1 representing more negative perceptions of social 
awareness and 5 representing more positive perceptions of social awareness. For each student, 
responses to the five Likert scale items were inclusively averaged (meaning all non-missing responses 
amongst the five items were averaged). 


The adjusted model results for grade 4 students show both the treatment (M = 3.84) and comparison 
group (M = 3.83) had positive leaning perceptions of social awareness. No treatment/comparison group 
differences in social awareness scores were found after two years of participating in the intervention. 
The grade 5 models showed similar positive perceptions of self-awareness for the treatment (M = 3.81) 
and comparison groups (M = 3.79). Although students in the treatment group exhibited social 
awareness scores that were .02-point higher than their counterparts in the comparison group after two 
years of participating in the intervention, this difference was not statistically significant. Figure 10 shows 
the social awareness outcome estimates. 


Figure 10. Social Awareness Outcomes Estimates 
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Reporting on Implications 


Meaningful and statistically significant differences in social and emotional competencies were not found 
between students who attended schools that received the Enhanced PSCM compared to their peers in 
schools that did not receive the Enhanced PSCM. However, students in treatment schools did have 
slightly higher self-management, growth mindset, and self-efficacy scores relative to their peers in 
control schools. Several factors could explain the lack of statistically significant findings. Specifically, 
variations in implementation of the Enhanced PSCM occurred across treatment schools. The nature of 
the Enhanced PSCM is that the approach is tailored to each school site. This means helping school sites 
identify where they need additional supports and then implementing those supports. The tailored 
approach to the intervention is thus impacted by differences in buy-in from site level staff, as well as 
different approaches at each school site. For example, site administrators selected the type of coaching 
and supports they wanted. In some cases, administrators wanted SCLs to demonstrate practices while 
other administrators wanted SCLs to co-facilitate strategies with the school staff. Once coaching was 
provided, school site staff could select how much they participated in coaching services provided by 
SCLs. 


An additional factor that could have contributed to the lack of program impacts is staff turnover at 
treatment school sites. For example, buy-in for the Enhanced PSCM was high for one school site. 
However, after a change in school site leadership and some key staff members, it was difficult to 
maintain momentum for the efforts being led by SCLs. In this case, there was a need to reestablish 
relationships between the SCL at the site and the new administrators at the site. 


Finally, SCLs varied their approaches to providing services under the Enhanced PSCM. Some coaches had 
set activities and curricula for their school sites. Others were more organic in their approach to 
coaching. Future studies should incorporate an Enhanced PSCM toolkit with structured activities and a 
menu approach to providing services. A menu approach would allow for consistent implementation of 
activities while also allowing for activities to be tailored to each school site. 


Despite the lack of significant findings, the direction of results is mostly consistent, indicating increases 
in SEL competencies for students receiving additional school climate supports. Future studies should 
continue to investigate the relationship between whole-school approaches to school climate and 
changes in SEL competencies for students. 
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